The positive and imperative effect of head tracking on simulating VR (Virtual Reality) or AR (Augmented Reality) or binaural audio rendering is a well-known phenomenon. In systems that implement such effects, head-tracking data is used in order to adjust the location of virtual objects (for example visual objects or sound sources) to a user's head movements in order to make them appear static in space rather than follow the user.
In a non-static environment, such as when the user is walking, riding a train, etc., the frame of reference of the user is constantly changing. In these situations, tracking head movements relative to a fixed reference becomes problematic and can lead to erroneous placement of virtual objects around the user.
Problems of head tracking in a non-static environment have been recognized in the conventional art and various techniques have been developed to provide solutions, for example:
United States Patent Publication No. US 2011/0293129 discloses a head tracking system that determines a rotation angle of a head of a user with respect to a reference direction, which is dependent on a movement of a user. Here the movement of a user should be understood as an act or process of moving including e.g. changes of place, position, or posture, such as e.g. lying down or sitting in a relaxation chair. The head tracking system according to the invention comprises a sensing device for measuring a head movement to provide a measure representing the head movement, and a processing circuit for deriving the rotation angle of the head of the user with respect to the reference direction from the measure. The reference direction used in the processing circuit is dependent on the movement of the user. The advantage of making the reference direction dependent on a movement of a user is that determining the rotation angle of the head is independent of the environment, i.e. not fixed to environment. Hence whenever the user is e.g. on the move and his body parts undergo movement the reference direction is adapted to this movement.
The reference(s) cited above teach background information that may be applicable to the presently disclosed subject matter. Therefore the full contents of these publications are incorporated by reference herein where appropriate for appropriate teachings of additional or alternative details, features and/or technical background.
General Description
In many cases of head tracking, the user's frame of reference also undergoes changes, for example in response to a change in the direction of linear motion of the user. For example, in a VR system for watching movies in airplane, simulating a cinema room with screen and speakers, and where the user is sitting. It is desired that the simulated cinema will remain stable relative to the user, rather than to the earth's ‘north’, such that when the airplane turns the cinema will remain in front of the user's eyes. Another example is listening to music with virtualized loudspeakers over headphones while walking and/or running. In this example it is desirable that the virtualized loudspeakers position will remain in front of the listener even when the user is turning a street corner.
The presently disclosed subject matter alleviates provides a method and system of head tracking for use in a non-static environment by adjusting the reference used for head tracking according to the movement of the user. Two or more inertial measurement units (IMU) or any other two or more sensors for acquiring location and/or movement and/or orientation data are used, in which at least one of the sensors is used for tracking orientation of the user's head, and at least another sensor is used for providing information indicating a possible need to adjust the frame of reference by which the head movements are tracked.
According to one aspect of the presently disclosed subject matter there is provided a system for providing head orientation relative to an adaptive reference orientation comprising a head sensor configured to provide head data describing a first head orientation associated with a user; one or more second sensors configured to provide second data; and a processing unit operatively coupled to the head sensor and the one or more second sensors and configured to: receive head data from the head sensor; receive second data from the one or more second sensors; adapt said adaptive reference orientation by moving it at least partly towards said first head orientation by an amount which is varied at least partly in accordance with said second data, thereby generating a new adaptive reference orientation; and generate, in accordance with said first head orientation and said new adaptive reference orientation, a second head orientation associated with the user as said first head orientation relative to said new adaptive reference orientation.
In addition to the above features, the system according to this aspect of the presently disclosed subject matter can comprise one or more of features (i) to (xii) listed below, in any desired combination or permutation which is technically possible:
(i) the amount is varied at least partly in accordance with changes in the statistical properties of said second data.
(ii) the adaptive reference orientation is adapted by rotating the adaptive reference orientation in accordance with an adjustment amount by interpolating between said adaptive reference orientation and said first head orientation. Interpolating can be performed using a quaternion spherical linear interpolation (“Slerp”) operation.
(iii) the second data can be used to compute or derive the adjustment amount by computing one or more measures based on the second data, and converting the one or more measures to the adjustment amount.
(iv) the one or more measures can be converted to the adjustment amount by comparing one or more of the one or more measures to one or more respective thresholds.
(v) the one or more measures can include one or more of relative stability, relative deviation, and/or absolute deviation.
(vi) the adjustment amount can be further processed for smoothing.
(viii) the adjustment amount can be further processed to control one or more of rise time, hold time, and decay time of the adjustment amount.
(ix) the second data includes data indicative of a need to change the adaptive reference orientation.
(x) the second data includes data indicative of at least one of: location, motion, orientation, velocity and acceleration associated with the user in at least one dimension of a three dimensional coordinate system.
(xi) binaural audio can be rendered in accordance with the second head orientation and delivered to headphones worn by the user.
(xii) at least one of virtual reality (VR) video and/or augmented reality (AR) video can be rendered in accordance with the second head orientation.
According to another aspect of the presently disclosed subject matter there is provided a computer implemented method of providing a head orientation relative to an adaptive reference orientation comprising, by a processing unit, receiving head data from a head sensor, the head data describing a first head orientation associated with a user; receiving second data from one or more second sensors; adapting the adaptive reference orientation by moving it at least partly towards said first head orientation by an amount which is varied at least partly in accordance with said second data, thereby generating a new adaptive reference orientation; and generating, in accordance with said first head orientation and said new adaptive reference orientation, a second head orientation associated with the user as said first head orientation relative to said new adaptive reference orientation.
This aspect of the disclosed subject matter can comprise one or more of features (i) to (xii) listed above with respect to the system, mutatis mutandis, in any desired combination or permutation which is technically possible.
According to another aspect of the presently disclosed subject matter there is provided a non-transitory program storage device readable by a computer, tangibly embodying computer readable instructions executable by the computer to perform a method of providing a head orientation relative to an adaptive reference orientation comprising, by a processing unit, receiving head data from a head sensor, the head data describing a first head orientation associated with a user; receiving second data from one or more second sensors; adapting the adaptive reference orientation by moving it at least partly towards said first head orientation by an amount which is varied at least partly in accordance with said second data, thereby generating a new adaptive reference orientation; and generating, in accordance with said first head orientation and said new adaptive reference orientation, a second head orientation associated with the user as said first head orientation relative to said new adaptive reference orientation.
This aspect of the disclosed subject matter can optionally comprise one or more of features (i) to (xii) listed above with respect to the system, mutatis mutandis, in any desired combination or permutation which is technically possible.
Among the technical advantages of the presently disclosed subject matter is the ability to perform head tracking in a non-static environment by using data from one or more second sensors operating independently of the head sensor to detect when the user's frame of reference has changed.
Among further advantages is the ability to continually generate an adaptive reference orientation for use in head tracking, thereby enabling the proper placement of virtual objects around the user's head in a non-static environment.